Parallel storage systems are widely used in many computing environments. Parallel storage systems provide high degrees of concurrency in which many distributed processes within a parallel application simultaneously access a shared file namespace.
Parallel computing techniques are used in many industries and applications for implementing computationally intensive models or simulations. For example, the Department of Energy uses a large number of distributed compute nodes tightly coupled into a supercomputer to model physics experiments. In the oil and gas industry, parallel computing techniques are often used for computing geological models that help predict the location of natural resources. One particular parallel computing application models the flow of electrons within a cube of virtual space by dividing the cube into smaller sub-cubes and then assigning each sub-cube to a corresponding process executing on a compute node.
Distributed parallel computing, however, is prone to failure as a failure of an individual compute node will interrupt the entire computation. Thus, fault tolerance techniques, such as checkpoint techniques, are often employed to store the current state of each process to persistent storage. After a failure, the application can then restart from the most recent checkpoint. The checkpoints are typically performed synchronously in time for each process.
There are a number of ways to organize the data streams when multiple processes simultaneously save data to a shared parallel storage system. For example, each process can create a single file across a set of different directories and then sequentially write a large amount of data to the single file. In a further implementation, a single process can create a shared file, and then all the processes write to the shared file in large segments that are aligned with block boundaries within the parallel file system.
When multiple processes create multiple small files concurrently, however, the performance of the parallel storage system will be impaired. Serialization can cause significant performance degradation as the parallel processes must remain idle while they wait for one another. Serialization is incurred when the parallel file system locks a shared object (such as a file or a directory) in order to maintain the consistency of the shared object. When multiple processes simultaneously try to create files in a shared directory, the directory must be locked so that multiple create operations on the directory do not collide.
Non-sequential access to the physical disk drives in the parallel storage system can also impair access pattern performance. When multiple small files are concurrently written, it is very challenging for the parallel file system to place the data into sequential regions of the disk. This type of access pattern can cause the disk to frequently switch the write position in a seek disk operation. Disk access patterns that incur many seek operations tend to perform an order of magnitude more poorly than patterns that do not incur seek operations.
A need therefore exists for improved techniques for storing multiple small files from a large number of clients concurrently accessing a shared storage system.